Steepest descent with momentum for quadratic functions is a version of the conjugate gradient method
نویسندگان
چکیده
منابع مشابه
Steepest descent with momentum for quadratic functions is a version of the conjugate gradient method
It is pointed out that the so called momentum method, much used in the neural network literature as an acceleration of the backpropagation method, is a stationary version of the conjugate gradient method. Connections with the continuous optimization method known as heavy ball with friction are also made. In both cases, adaptive (dynamic) choices of the so called learning rate and momentum param...
متن کاملSteepest Descent and Conjugate Gradient Methods with Variable Preconditioning
We analyze the conjugate gradient (CG) method with variable preconditioning for solving a linear system with a real symmetric positive definite (SPD) matrix of coefficients A. We assume that the preconditioner is SPD on each step, and that the condition number of the preconditioned system matrix is bounded above by a constant independent of the step number. We show that the CG method with varia...
متن کاملNonsymmetric Preconditioning for Conjugate Gradient and Steepest Descent Methods1
We numerically analyze the possibility of turning off postsmoothing (relaxation) in geometric multigrid when used as a preconditioner in conjugate gradient linear and eigenvalue solvers for the 3D Laplacian. The geometric Semicoarsening Multigrid (SMG) method is provided by the hypre parallel software package. We solve linear systems using two variants (standard and flexible) of the preconditio...
متن کاملOn the steepest descent algorithm for quadratic functions
The steepest descent algorithm with exact line searches (Cauchy algorithm) is inefficient, generating oscillating step lengths and a sequence of points converging to the span of the eigenvectors associated with the extreme eigenvalues. The performance becomes very good if a short step is taken at every (say) 10 iterations. We show a new method for estimating short steps, and propose a method al...
متن کاملA Generalization of the Steepest Descent Method for Matrix Functions∗
We consider the special case of the restarted Arnoldi method for approximating the product of a function of a Hermitian matrix with a vector which results when the restart length is set to one. When applied to the solution of a linear system of equations, this approach coincides with the method of steepest descent. We show that the method is equivalent to an interpolation process in which the n...
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
ژورنال
عنوان ژورنال: Neural Networks
سال: 2004
ISSN: 0893-6080
DOI: 10.1016/s0893-6080(03)00170-9